Assessing the Learning and Transfer of Data Collection Inquiry Skills Using Educational Data Mining on Students’ Log Files

In this paper we explored whether engaging in two inquiry skills associated with data collection, designing controlled experiments and testing stated hypotheses, within microworlds for one physical science domain (density) impacted the acquisition of inquiry skills in another domain (phase change). To do so, we leveraged educational data mining techniques to both assess and estimate students’ inquiry skills across domains. Analyses revealed that honing these skills in density activities provided benefits in terms of transfer and skill acquisition. More specifically, students who practiced in density activities first were more likely to show mastery of the designing controlled experiments skill than those who had no prior practice. These same students were also more likely to test their stated hypotheses during their first data collection in phase change. Thus, practice in one domain can positively impact acquisition and transfer of skill in a second domain, suggesting that inquiry skills also have a degree of domain generality. Introduction Science educators agree that cultivating inquiry skills is critical for students to become scientifically literate (National Research Council, 1996, 2000, 2011; Kuhn, 2005a). However, typical standardized science tests do not adequately reflect or assess complex inquiry process skills (Quellmalz, Timms & Schneider, 2009). Performance assessments of inquiry, instead, have been argued to be better-suited for this purpose (cf. Black, 1999; Pellegrino, 2001). Devising scalable and reliable performance assessments, though, is difficult for two reasons. First, it is difficult to separate inquiry skills from content understanding (Mislevy, Steinberg & Almond, 2002; Mislevy, et al., 2003). Second, inquiry processes are mutli-faceted, and there is no one single “right or wrong” way to engage in science inquiry (Shute, Glaser & Raghavan, 1989; Glaser, Schauble, Raghavan, & Zeitz, 1992). Given inquiry’s importance, proper techniques for measuring inquiry are needed. It is also important to better understand inquiry learning so that we can foster transfer of such skills to novel tasks (Kirschner, Sweller & Clark, 2006; Hmelo-Silver, Duncan & Chinn, 2007). Regarding transfer, it has been suggested that inquiry skills are tightly tied to the domain in which they are learned (van Joolingen, de Jong & Dimitrakopoulout, 2007), but some evidence exists that long-term, repeated practice of inquiry (Kuhn, Schauble & Garcia-Mila, 1992; Dean Jr. & Kuhn, 2006; Kuhn & Pease, 2008), and scaffolding or teaching these skills explicitly (Klahr & Nigam, 2004) can lead to successful acquisition and transfer to novel tasks. In the present paper, we address two goals. First, we describe our approach for developing reliable, scalable performance measures of inquiry. Second we leverage those assessment techniques to examine how inquiry skills transfer between two physical science domains. We focus on on two inquiry skills, designing controlled experiments and testing stated hypotheses. Designing controlled experiments entails selecting experiments to yield data that supports determining the effects of manipulable variables on outcomes. Testing stated hypotheses refers to generating data with the intention to support or refute a specific hypothesis. These skills are measured as students conduct inquiry within microworlds for two domains, phase change and density, developed within the Science Assistments system (Gobert et al., 2007; Gobert et al., 2009). In our approach, we leverage techniques from Educational Data Mining (cf. Baker & Yacef, 2009; Romero & Ventura, 2010) to assess and track inquiry skills across several activities within each domain. To assess these skills, we use validated detectors (models) of students’ inquiry behaviors that were constructed based on student log files (Sao Pedro et al., 2010, in press). We then produce estimates of student proficiency for each skill by aggregating all assessments into a Bayseian Knowledge Tracing model (Corbett & Anderson, 1995). This approach is rigorous because an EDM affords the ability to estimate how well the models assess and track skill. Furthermore, these can be done in real time. Thus, we argue, the approach is scalable, and could provide a possible model for inquiry assessment. These techniques were also leveraged to measure whether the two skills of interest, designing controlled experiments and testing stated hypotheses, transfer across two physical science domains. These two domains are Density and Phase Change. More specifically, we analyzed whether students who practiced in density activities first had a greater likelihood of demonstrating the skills or reaching mastery than students with no prior practice. With our assessment and skill tracking models, we can assess this transfer in a finer-grained way than other prior studies of inquiry (e.g. Kuhn & Pease, 2008). The remainder of this paper is organized as follows. First, we describe the two skills of interest in more detail, and present related work on assessing data collection skills. We then present a high-level view of our approach for assessing and estimating proficiency at these data collection skills using our educational data mining techniques. Next, we present our results on the transfer of these skills between domains which leveraged our assessment and estimation techniques. Finally, we present a discussion and conclusions of our paper.